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ABSTRACT 



This paper recounts the use of a computerized 
multiple-measures placement model at Yuba College in Marysville, California. 
Yuba began using computerized placement tests in 1994, based on suggestions 
from faculty and student assessment staff. Two years later, a set of multiple 
measures criteria, consisting mainly of self-reported high school history was 
selected to weigh scores in placement recommendations. In spring 1997, new 
placement criteria were implemented for students testing for fall enrollment, 
and all placement recommendations were made by the computerized system. By 
fall 1997, student success among those placed by the test was above 70% in 
many basic skills courses. The placement accuracy ratio increased to more 
than .80. Conclusions of this pilot study show that carefully selected 
multiple measures applied by a computer to course placement significantly 
increases placement accuracy. Students whose placements were adjusted through 
the use of behavioral and historical characteristics were more accurately 
placed and succeeded at a higher rate than students placed on the basis of 
test scores alone, and at a much higher rate than those placed by subjective 
application of undefined multiple measures by counselors. Attached are a 
cover memo to faculty, and faculty and student surveys. (AS) 
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Introduction 



Couree placement has been determined by most researchers to be one of the major keys to 
student retention, persistence, and success. Learning takes place when students have the tools 
with which to learn and are placed in an environment which is conducive to processing 
information. Students who possess academic skills adequate for the learning environment their 
course presents are highly likely to succeed in the course. Students who succeed in their initial 
courses are more likely to persist than students who fail their first college coxuses. Today’s 
yoimg students would call it a “no-brainer,” then, that assessing academic skills so as to place 
students at the highest level in which they are likely to succeed will promote increased retention 
and persistence. 

California Community Colleges are mandated to use “multiple measures” for course placement. 
Placement examinations must be approved by a committee according to regulations, and tests 
must be validated for disproportionate impact content validity, and either criterion or 
consequential validity. With a limited number of approved tests available, computerized testing 
has gained momentum. Currently, 23 community colleges are using a form of computerized 
testing, with a dozen more in various stages of implementation. 

With computerized testing comes the ability to expand simple test administration into 
computerized placement management, and fi’om there to data collection for multiple measures 
placement. This paper presents the results of a computerized multiple-measures placement 
model. Information includes: choosing acceptable alternate measures, assigning weight to 
various measurements, and the results in terms of student success and placement accuracy. 



In the Beginning There was Chaos 

In 1994, Yuba College began using computerized placement tests. The decision to use 
ACCUPLACER was based on faculty preference and the experience of the assessment staff with 
both computerized products. Cut scores were set using a make-shift concordance table 
developed fix)m the APS test used previously. Placement recommendations were reviewed and, 
if appropriate, over-ridden by counselors using xmdefined criteria. Prior to beginning the 
computerized testing program, student success rates ranged fi’om 45 to 60 percent in basic skills 
coiuses, with retention hovering well under 70 percent. 

Cut scores were reviewed and adjusted after one semester using final course grades as the 
dependent variable. Adjustments were minor, usually less than five points on a 120 point scale. 
Overall student success rates improved slightly, but success rates among those students who had 
followed placement recommendations improved significantly. 

In 1996, a set of multiple measures criteria consisting mainly of self-reported high school history 
was selected to weight scores in placement recommendations. By fall, 1997 student success 
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among those placed by the placement test was above 70% in many basic skills courses. The 
placement accuracy ratio increased to more than .80. Placement accuracy, to be defined later, 
accounts for both alpha and beta errors in placement. 



Multiple Measures Criteria 

California regulations do not define multiple measures except to say they cannot include another 
test. Regulations also do not specify how many measures must be used nor how they must be 
applied. Selecting which measures to use can become time-consuming and create controversy 
among participants. Counselors and teaching faculty, although their goals are similar, have 
different perspectives on what constitutes valid measures. It can be extremely difficult to limit 
the number of measures available for use, and even more difficult to select those to be included. 

Three criteria, consisting only of high school performance, were initially set up as multiple 
measures criteria and weighted to influence the placement of students whose test scores were 
near the cutoff After several failed attempts to establish limits and identify acceptable criteria, 
meetings were held with department faculty in early Spring, 1997. At each meeting, the 
operations of the computerized placement system were explained and the limitations were 
stressed. Manageability of the system was explained. A placement rule must be written for each 
level of placement for each possible combination of responses to measurement criteria. For 
example, five questions with five responses each would require 3125 rules for each level of 
placement. At Yuba College, that would mean 15,625 rules each for math and English 
placement: an unmanageable situation even without considering reading and ESL classes. 

Faculty were then asked to identify measures their experience had taught them influenced student 
success. Using a modified affinity process, four to five criteria were selected as the most 
infl uential on student success. Once begun, the entire process took less than an hour to complete 
in each department. 

After the criteria had been selected, the relative importance of each was discussed and tentative 
weight was assigned to each possible response. Experience had taught that negative weight 
should be limited to only those criteria which might cause a student with adequate academic 
skills to fail, such as number of units, time available for study, and hours worked. Historic data, 
such as highest level of study, may influence success beyond that predicted by test scores, but 
seldom predict failure below the skills demonstrated in the test. Faculty were cautioned against 
making the system unmanageable with too many possible combinations of responses, and 
assigning too much weight to any one criterion. When the discussions were ended, language 
faculty had chosen five criteria, two historic and three behavioral, with a total weight range from 
.95 to 1.17. Math faculty had chosen four criteria, two each, with a total weight range from .95 
to 1.14. By combining response choices into meaningful groups, possible combinations were 
limited to 144 for each level of placement: a lot of work, but manageable. 
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Pilot Study 



In Spring, 1997, the new placement criteria were implemented for students testing for Fall 
enrollment. Placement over-rides were allowed with counselor approval, but counselors were 
encouraged to have students access the challenge process if they were dissatisfied with their 
placements. All placement recommendations were made by the computerized system. 

Counselor over-rides accounted for about 15% of all course placements. At transferable levels, 
nearly 70% of students were placed by having taken a prerequisite course. In the developmental 
courses, of course, prerequisite placement was not a factor. 

Among those students placed by over-ride, fewer than 40% completed the assigned course with a 
“C” or better. About half dropped the course, and the rest received “D” or “F” grades. The 
success rate for those students in courses by virtue of prerequisites averaged 67%, with success 
defined as completing the course with a “C” or “CR” or better. Several courses had lower 
success rates, and faculty examined and revised course outlines accordingly. The success rates 
among students placed by the computerized placement system ranged fi'om 61% in a math class 
to 74% in an English course. This was marked improvement fi'om 1994 data. The graphs in the 
attached presentation materials indicate the change. One notable exception was a decrease in 
student success in the vocational level English course. Investigation determined that the course 
outline had changed and that, in fact, the exit criteria for the course had been raised substantially. 
The cut score was immediately adjusted, and the problem corrected in the subsequent term. 

Due to a substantial number of withdrawals which could not be attributed to failures and the 
substantial grade variances among faculty teaching the same courses, there emerged a need to 
identify another means to validate the placement system. Students drop courses for any number 
of reasons not related to their probability of succeeding in the course. Since drops are not 
necessarily attributable to the placement system, they must be removed fi'om the sample before 
analysis. This substantially reduces the sample size and brings into question sample vaUdity and 
reliability. Although research indicates that perception of failure is not a substantial reason for 
students to drop classes, eliminating those students fi'om the analysis leaves the appearance of a 
contaminated sample. Grade variance, of course, always creates noise in the data set and distorts 
placement research results. The result was the development of “Placement Accuracy,” a method 
of assessing the placement system rather than just student outcomes. Credit for the initial model 
for this method goes to Darlene Nold and her associates at Aimes Community College, Greeley, 
Colorado. 



Placement Accuracy 

Setting a cut score requires making two erroneous assumptions. First, we assume that everyone 
who scores at or above the cut score will pass the course. Second, we assume that anyone who 
scores below the cut score would fail the course if admitted. Neither, of course, is absolutely 
true. The best we can do is place the cut score where errors will be minimized. We can argue 
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whether we prefer inclusive errors (alpha), where we admit students who will fail the course; or 
exclusive errors (beta), where we exclude students who would pass the course if admitted. 
Preliminary extensions of the research discussed here indicate that when placement errors are at 
their minimum, alpha errors will be higher than beta errors. Placement accuracy is computed by 
factoring both alpha and beta errors into the success equation. 

Surveys was distributed to all faculty and all students in the subject classes after the fourth week, 
but before mid-term. Forms are attached. The faculty survey was a simple class roster which 
asked for each student’s level of preparation for the materials in the course on a 3-point scale. 
Students were “under-prepared,” “adequately prepared,” or “over-prepared.” In a second 
column, faculty were asked to indicate each student's likelihood of earning a “C” or better based 
on ability alone (Highly-likely, Probable, Highly-unlikely). The latter indicator was used for a 
validity check. For students who had dropped the course, faculty were asked, if they could, to 
evaluate whether the student was passing at the time of the drop, assuming that passing meant 
adequate preparation. 

The student survey asked for the anticipated grade for the course, perception of placement 
accuracy (placed properly, too low, or too high for ability), and the difficulty of the course 
(moderately difficult, easy, extremely difficult). Definitions of terms were provided on the 
survey form. Faculty were asked to distribute the survey to students after they had made their 
evaluations, and to collect the surveys and return them to the assessment center. We received a 
100% response rate firom faculty, and 76% firom students. We did not test the sigmficance of 
absence on the day the surveys were distributed. 

Student responses were highly correlated with faculty responses regarding placement accuracy, 
as well as anticipated grade. Later tests also found students’ anticipated grades highly correlated 
with actual grades. Faculty likelihood of a “C” or better assessment correlated with actual grade, 
but only on a dichotomous basis. Grade variance among instructors probably accounted for the 
inability to find a correlation with the grade scale. 

Survey results were separated by course, and appropriate test scores and placement levels were 
added to the data file. Since placement over-rides were allowed and commonly used by 
counselors, a significant number of students were enrolled in courses above their placement 
levels. The sample was sorted by placement method: placement examination, over-ride, and 
prerequisite course. Placement accuracy was determined by dividing the number of students 
rated adequately prepared who were at their placement levels (S) plus the number of students 
rated under-prepared who were above their placement levels (U) by the total course enrollment 
(E): ((S+U)/E). This method counts those students who were enrolled above their placement 
levels and rated adequately prepared (A) as failures of the placement system, along with those 
who were rated over-prepared (O) and those enrolled at their placement levels and rated under- 
prepared (P). The following table illustrates the placement accuracy model. 






A 



Placement \ Enrollment 


At Placement Level 


Above Placement Level 


Accurate 


Adequately Prepared 


Under-Prepared 


Inaccurate 


Under-Prepared 

Over-Prepared 


Adequately Prepared 
Over-Prepared 



A few students were actually enrolled below their placement levels. This, of course, was the 
result of personal choice, and unless they had been rated under-prepared for their courses (none _ 
was) there was not a means to evaluate their placement accuracy. Their records were discarded. 

Across all evaluated courses, the placement accuracy ratio from the placement exam and 
computerized multiple-measures placement system was .78. High ranged to .84, and low was .71 
for the placement system. Prerequisite courses had a placement accuracy ratio of .68, and over- 
rides .38. Using scores alone, without the multiple measures factors, the test had an accuracy 
ratio of .73 overall. Among those students who had been placed by counselor over-ride, nearly 
all of those who were rated adequately prepared would have been placed in the course with the 
revised multiple measures placement system. 



Conclusions 

Carefully selected multiple measures applied by a computer to course placement increases 
placement accuracy significantly. Students whose placements were adjusted through the use of 
behavioral and historical characteristics were more accurately placed and succeeded at a higher 
rate than students placed on the basis of test score alone, and at an extremely higher rate than 
those placed by subjective application of undefined multiple measures by counselors. Students 
succeeded in their initial placement courses at a higher rate as a result of computerized multiple- 
measures placement. 

Adjustments made to placements on the basis of a limited number of clearly defined measurable 
characteristics are valid for placing students into a proper learning environment. Adding other 
characteristics to the equation does not seem to improve the accuracy of placements, especially if 
those characteristics are added subjectively. Given that proper placement improves retention 
and persistence, factoring historical and behavioral characteristics into placement 
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recommendations offers significant improvement to the learning environment for those students 
affected. 



Need for Further Research 

More research is needed in tracking the use of various non-test placement measures into long- 
term student outcomes. Research questions include: Do students placed by a multiple measures 
system differ significantly from students self-placed or those placed on the basis of test alone in 
their success rates in subsequent courses; retention, persistence, transfer, graduation rates; and 
other measures of student success? Do students from classes in which all students were placed 
by a multiple-measures and prerequisite placement system differ significantly from other 
students on the basis of their performance in subsequent courses and other outcomes 
measurements? Which student characteristics that are collectable on a self-reported basis are 
valid for use in multiple-measures placement schemes? 
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Memorandum 

Yuba Community College District 
Office of Research & Assessment 



Room 113 
Phone 741-6864 



Date: 


January 7, 1999 


To: 


English Faculty 


From: 


Dr. Ron Gordon 


Subject: 


Placement Evaluation 



Copy: Jay Drury 

Attached are rosters for your current English classes. On each are spaces for you to evaluate how 
accurately each student was placed in your course. This information will be used to readjust cut 
scores for English placement and to fine-hme the placement system. Please evaluate first how 
accurately you believe the system was if it placed the student in your course. The student was 
placed either too low, meaning he or she should have been placed in the next higher level course; 
too high, meaning she or he should have been placed at a lower level, or accurately, meaning the 
student has a high probability of succeeding in your class. PLEASE BASE YOUR 
EVALUATION ON THE STUDENTS’ ABILITY ONLY, NOT ON ATTENDANCE, 
MOTIVATION, OR OTHER FACTORS WHICH AFFECT GRADES. This survey is to 
evaluate the placement system and its ability to place students according to their abilities. 

Second, please indicate your impression of how likely each student, based on ability alone, 
would be to earn a “C” or better in your class. When you have finished, please return the forms 
to Research & Assessment, or ask Pat to put them in my mailbox. As soon as I get all the forms 
back. I’ll evaluate the placement results, then meet with department faculty to discuss revising 
placement criteria. I realize that completing these forms thoughtfully will take some time, and I 
appreciate your patience. I believe it is the only way we can establish cut scores that will help 
students get to the classes that will make them successful. Please make every effort to evaluate 
fairly and to return the forms in a timely maimer. 

Thank you for your help. 
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Faculty Survey for Placement Validation 

Course: Math 50 Prerequisite: Math 110/111 

Term: Fall, 1997 Class Code: ##### 

In the space labeled “PREPARED” after each student’s name, please indicate whether the 
student is: 1 = Under-prepared for the class (does not possess adequate skills to comprehend the 
new materials presented in this class), 2 = Adequately prepared (possess the skills to complete 
the course successfully with a normal amount of study and tutoring), 3 = Over-prepared (Highly 
likely to succeed in the next higher course). In the space labeled “SUCCESS”, please indicate 
the likelihood that each student will earn a grade of “C” or better based on ability alone: 1 = 
Highly likely, 2 = Probable, 3 = Highly unlikely. 

This information will be used to help evaluate and validate the placement cut scores and the 
placement system as required by regulations. Please reply promptly by returning this survey to 
the Office of Research and Assessment, Room 114. 

SSN NAME PREPARED SUCCESS 

This area contains the class roster. 




II 



Yuba Community College District 
Student Placement Survey 



Course: English 5 1 



Please answer each question below by selecting the answer which most closely agrees with your 
placement in this course. This information will be kept confidential and will not affect your 
grade or other activities related to this course. 



How well do you believe you were prepared for the work in this course? 
Not prepared, Adequately prepared, Over-prepared 



Do you find the usual assignments in this course to be: 

Too difficult? Within your capabilities? Too easy (already know the material)? 



What grade do you believe you will earn in this course? 
A B C D F 
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